Repository navigation
Stop the zarr drain when the tracked ref moves - #1132
Merged
Merged
Conversation
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
setup()resets the Hallu clone toorigin/$ZARR_DRIVER_REFon every run, which is what makes the Python driver self-deploying (ADR 0029). That property silently does not hold during a backfill, because a run does not end until the queue is empty: the drain claims dataset after dataset inside one invocation while holding the flock, and every hourly tick meanwhile logsanother hallu-zarr instance holds the lock; exitingwithout ever reachingsetup().Closes #1129.
This is not hypothetical
Measured on Hallu, roughly two days after the cutover to the nemar-cli clone:
That run had been holding the lock for 1 day 23 hours. The driver it was executing had ADR 0027 discovery, the #1068 gate and streaming, but no
apply_worker_mem_limit, norss_trusted, and no MaxShield -- so two days of conversions ran without the entire epic #1108 memory work, including #1110, where an OOM-killed worker takes the whole dataset down and burns its retry budget. That is the exact failure epic #1108 was written to remove.The existing drift guard could not catch this. It compares the running script against
$DRIVER_REPO/scripts/zarr/$(basename $0), and a stale run'sDRIVER_REPOis the same stale clone it was launched from: self-consistent, and two deploys behind. It answers "is this script stale relative to the repo it was launched from", not "has that repo moved since".Change
ref_moved()comparesgit ls-remote origin $DRIVER_REFagainst the clone's HEAD, checked between datasets in the drain loop. On a move the run logs and breaks, and the next:30tick redeploys.One
ls-remoteper dataset is noise against per-dataset conversion times measured in minutes, and the queue persists across runs, so stopping costs nothing but the tick.Failing safe was the design constraint. A network blip must never halt hours of backfill, so an unresolvable ref or an unreachable remote returns "not moved" and the drain continues; the worst case is that the redeploy waits for the next dataset boundary.
Test plan
There is no harness for this script, so
ref_moved()was exercised directly against a real local git remote, all four paths:DRIVER_REFdoes not resolvebash -nclean;shellcheck -S warningclean on the whole script.AGENTS.md said the driver redeploys "every run", which was the assumption that hid this; it now says what actually happens.